An effective web document clustering for information retrieval

The size of web has increased exponentially over the past few years with thousands of documents related to a subject available to the user. With this much amount of information available, it is not possible to take the full advantage of the World Wide Web without having a proper framework to search through the available data. This requisite organization can be done in many ways. In this paper we introduce a combine approach to cluster the web pages which first finds the frequent sets and then clusters the documents. These frequent sets are generated by using Frequent Pattern growth technique. Then by applying Fuzzy CMeans algorithm on it, we found clusters having documents which are highly related and have similar features. We used Gensim package to implement our approach because of its simplicity and robust nature. We have compared our results with the combine approach of (Frequent Pattern growth, K-means) and (Frequent Pattern growth, Cosine_Similarity). Experimental results show that our approach is more efficient then the above two combine approach and can handles more efficiently the serious limitation of traditional Fuzzy C-Means algorithm, which is sensitive to initial centroid and the number of clusters to be formed. Keywords-Clustering, Fuzzy C-Means, Frequent Pattern growth, Gensim, Vector space model